Theoretically Principled Deep RL Acceleration via Nearest Neighbor Function Approximation

Recently, deep reinforcement learning (RL) has achieved remarkable empirical success by integrating neural networks into RL frameworks. However, these algorithms often require a large number of training samples and admit little theoretical understanding. To mitigate issues, we propose theoretically principled nearest neighbor (NN) function approximator that can replace the value in methods. Inspired human similarity judgments, NN estimates action values using rollouts on past observations provably obtain small regret bound depends only intrinsic complexity environment. We present (1) Nearest Neighbor Actor-Critic (NNAC), an online policy gradient algorithm demonstrates practicality combining approximation with RL, (2) plug-and-play update module aids existing Experiments classical control MuJoCo locomotion tasks show NN-accelerated agents achieve higher sample efficiency stability than baseline agents. Based its benefits, believe be further applied to other complex domains speed-up learning.